{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/MisaOgura/flashtorch/blob/master/examples/activation_maximization_colab.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activation maximization\n",
    "\n",
    "---\n",
    "\n",
    "A quick demo of activation maximization with [FlashTorch 🔦](https://github.com/MisaOgura/flashtorch), using the pre-trained VGG16 model.\n",
    "\n",
    "\n",
    "❗This notebook is for those who are using this notebook in **Google Colab**.\n",
    "\n",
    "If you aren't on Google Colab already, please head to the Colab version of this notebook **[here](https://colab.research.google.com/github/MisaOgura/flashtorch/blob/master/examples/activation_maximization_colab.ipynb)** to execute.\n",
    "\n",
    "---\n",
    "\n",
    "[Activation maximization](https://pdfs.semanticscholar.org/65d9/94fb778a8d9e0f632659fb33a082949a50d3.pdf) is one form of feature visualization that allows us to visualize what CNN filters are \"looking for\", by applying each filter to an input image and updating the input image so as to maximize the activation of the filter of interest (i.e. treating it as a gradient ascent task with filter activation values as the loss).\n",
    "\n",
    "The optimization and visualization is available via `flashtorch.activmax.GradientAscent`. The implementation is inspired by [this demo](https://blog.keras.io/category/demo.html) by Francois Chollet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0. Set up\n",
    "\n",
    "A GPU runtime is available on Colab for free, from the `Runtime` tab on the top menu bar.\n",
    "\n",
    "It is **highly recommended to use GPU** as a runtime for the enhanced speed of computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install flashtorch\n",
    "\n",
    "!pip install flashtorch torch==1.5.0 torchvision==0.6.0 -U"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "import torchvision.models as models\n",
    "\n",
    "from flashtorch.activmax import GradientAscent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Load a pre-trained Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = models.vgg16(pretrained=True)\n",
    "\n",
    "# Print layers and corresponding indicies\n",
    "\n",
    "list(model.features.named_children())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Specify layers and filters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conv1_2 = model.features[2]\n",
    "conv1_2_filters = [17, 33, 34, 57]\n",
    "\n",
    "conv2_1 = model.features[5]\n",
    "conv2_1_filters = [27, 40, 68, 73]\n",
    "\n",
    "conv3_1 = model.features[10]\n",
    "conv3_1_filters = [31, 61, 147, 182]\n",
    "\n",
    "conv4_1 = model.features[17]\n",
    "conv4_1_filters = [238, 251, 338, 495]\n",
    "\n",
    "conv5_1 = model.features[24]\n",
    "conv5_1_filters = [45, 271, 363, 409]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Optimize and visualize filters\n",
    "\n",
    "Creating an instance of `GradientAscent` class with the model _without fully-connected layers_ allows us to use flexible input image sizes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g_ascent = GradientAscent(model.features)\n",
    "\n",
    "g_ascent.use_gpu = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By calling the `visualize` method and passing in the layer and filter indeciies defined above, it performs optimization and visualization.\n",
    "\n",
    "This is perhaps the most common way to use the `GradientAscent` class, but there are other APIs available according to your use cases (see section 4)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g_ascent.visualize(conv1_2, conv1_2_filters, title='conv1_2');\n",
    "g_ascent.visualize(conv2_1, conv2_1_filters, title='conv2_1');\n",
    "g_ascent.visualize(conv3_1, conv3_1_filters, title='conv3_1');\n",
    "g_ascent.visualize(conv4_1, conv4_1_filters, title='conv4_1');\n",
    "g_ascent.visualize(conv5_1, conv5_1_filters, title='conv5_1');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that, in the earlier layers (conv1_2, conv2_1), filters get activated by colors and simple patterns such as virtical, horisontal and diagonal lines.\n",
    "\n",
    "In the intermediate layers (conv3_1, conv4_1), we start to see more complex patterns.\n",
    "\n",
    "Then oncepts like 'eye' (filter 45) and 'entrance (?)' (filter 271) seem to appear in the last layer (conv5_1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Other ways to use `GradientAscent`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4-1. `GradientAscent.visualize`: randomly select filters\n",
    "\n",
    "If you have a convolutional layer you want to vizualise, but you don't know which filters to choose, you can just pass in the layer to `visualize` without `filter_idxs`. It will randomly choose filters. You can adjust the number of filters chosen by passing `num_subplots` (default=4)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g_ascent.visualize(conv5_1, title='Randomly selected filters from conv5_1');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4-2. `GradientAscent.visualize`: plot one filter\n",
    "\n",
    "If you just want to visualize one filter, you can do so by specifying the filter index as an integer, not a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g_ascent.visualize(conv5_1, 3, title='conv5_1 filter 3');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4-3. `GradientAscent.visualize`: return image tensor\n",
    "\n",
    "If you want to grab the optimized image data, set `return_output` to `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = g_ascent.visualize(conv5_1, 3, title='conv5_1 filter 3', return_output=True);\n",
    "\n",
    "print('num_iter:', len(output))\n",
    "print('optimized image:', output[-1].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4-4. `GradientAscent.deepdream`: create DeepDream\n",
    "\n",
    "You can create a [DeepDream](https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html) by supplying a path to your own image.\n",
    "\n",
    "This will optimize the supplied image, instead of a random input noise, with regards to the filter specidied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the example image\n",
    "\n",
    "!mkdir -p images\n",
    "\n",
    "!wget https://github.com/MisaOgura/flashtorch/raw/master/examples/images/jay.jpg -P /content/images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g_ascent.deepdream('/content/images/jay.jpg', conv5_1, 33)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4-5. `GradientAscent.optimize`: perform optimization only (no visualization)\n",
    "\n",
    "If no visualization is needed, or if you want to futher customize visualization, you can call the `optimize` method directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = g_ascent.optimize(conv5_1, 3)\n",
    "\n",
    "print('num_iter:', len(output))\n",
    "print('optimized image:', output[-1].shape)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
